{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 如何使用千帆 Python SDK 搭配预置大模型服务进行批量推理\n",
    "\n",
    "在进行模型评估或其他任务时，通常需要对大量数据进行预测。然而，模型推理过程往往耗时较长，通过循环串行执行会增加整体时间成本，而并行执行则需要额外的开发工作。\n",
    "\n",
    "SDK 提供了多种解决方案来应对这一场景，其中包括：\n",
    "\n",
    "- [本地并行推理](https://github.com/baidubce/bce-qianfan-sdk/blob/main/cookbook/batch_prediction.ipynb)：利用 SDK 内置的批量推理功能，在本地通过并行调用模型接口实现高效的批量预测。\n",
    "- [数据集评估](https://github.com/baidubce/bce-qianfan-sdk/blob/main/cookbook/dataset/batch_inference_using_dataset.ipynb)：利用 SDK 的 Dataset 模块，调用平台提供的数据集评估功能，以便快速而有效地完成任务。\n",
    "- [离线批量推理](https://github.com/baidubce/bce-qianfan-sdk/blob/main/cookbook/offline_batch_inference.ipynb)：对于时间要求不那么严格的场景，可以考虑利用平台提供的离线批量预测能力，以降低实时推理的负载压力。\n",
    "\n",
    "本文将介绍第二种解决方案，即使用 Dataset 进行评估，在 0.2.8 版本中，千帆 Python SDK 增加了对该功能的支持，使用该功能需要视情况开通千帆的模型服务，确保您的账号可以调用您想进行批量推理的服务。\n",
    "\n",
    "# 准备工作\n",
    "\n",
    "在开始之前，请确保你的千帆 Python SDK 已经升级到了 0.2.8 及以上版本。\n",
    "nest_asyncio 是一个异步库，用于支持 Python SDK 的异步推理功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "#-# cell_skip\n",
    "! pip install -U \"qianfan>=0.2.8\"\n",
    "! pip install nest_asyncio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "并且在环境变量中设置好 Access Key 与 Secret Key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-02-23T07:49:30.708585Z",
     "start_time": "2024-02-23T07:49:30.696228Z"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "\n",
    "from qianfan.utils import enable_log\n",
    "\n",
    "# os.environ['QIANFAN_ACCESS_KEY'] = 'your_access_key'\n",
    "# os.environ['QIANFAN_SECRET_KEY'] = 'your_secret_key'\n",
    "\n",
    "os.environ[\"QIANFAN_QPS_LIMIT\"] = \"1\"\n",
    "os.environ['QIANFAN_LLM_API_RETRY_COUNT'] = \"3\"\n",
    "\n",
    "# 选择打印出来的日志等级，目前打印出 info 级别\n",
    "enable_log(logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正文\n",
    "\n",
    "为了开始批量推理，我们首先需要获取到用于做批量推理输入的数据集文件，并且指定用做推理输入的列名"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-02-23T07:49:32.731457Z",
     "start_time": "2024-02-23T07:49:32.692145Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [02-23 15:49:32] dataset.py:489 [t:8596849280]: no data source was provided, construct\n",
      "[INFO] [02-23 15:49:32] dataset.py:358 [t:8596849280]: construct a file data source from path: data_file/qa_pair.csv, with args: {'input_columns': ['prompt'], 'reference_column': 'response'}\n",
      "[INFO] [02-23 15:49:32] file.py:165 [t:8596849280]: use format type FormatType.Csv\n",
      "[INFO] [02-23 15:49:32] dataset.py:934 [t:8596849280]: list local dataset data by 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'prompt': '地球的自转周期是多久？', 'response': '大约24小时'}\n"
     ]
    }
   ],
   "source": [
    "from qianfan.dataset import Dataset\n",
    "\n",
    "dataset_file_path = \"data_file/qa_pair.csv\"\n",
    "dataset_input_column_list = [\"prompt\"]\n",
    "\n",
    "# 预期输出列列名，当数据集为对话类数据集时必填，为非对话数据集时选填。\n",
    "# 对应列的数据会在推理结果中出现\n",
    "reference_column = \"response\"\n",
    "\n",
    "ds = Dataset.load(data_file=dataset_file_path, input_columns=dataset_input_column_list, reference_column=reference_column)\n",
    "\n",
    "# 预览数据格式\n",
    "print(ds.list(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在导入之后，用户可以根据自己的需求，传入不同的参数来使用不同的方式进行推理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-02-23T07:07:43.381660Z",
     "start_time": "2024-02-23T07:03:55.104609Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13072834560]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13089624064]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13139992576]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13106413568]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13156782080]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13123203072]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13173571584]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13190361088]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13257519104]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13274308608]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13207150592]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13240729600]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13391835136]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13223940096]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13307887616]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13408624640]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13324677120]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13291098112]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13375045632]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13358256128]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13341466624]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13425414144]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13492572160]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13442203648]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13458993152]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13542940672]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13576519680]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13559730176]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13475782656]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13626888192]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13660467200]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13509361664]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13526151168]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13677256704]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13610098688]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13643677696]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13593309184]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13694046208]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13710835712]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13777993728]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13744414720]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13794783232]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13811572736]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13727625216]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13761204224]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13828362240]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13861941248]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13845151744]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13895520256]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13912309760]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13929099264]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13962678272]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13945888768]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13878730752]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13979467776]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:13996257280]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14029836288]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14013046784]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14046625792]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14113783808]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14096994304]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14063415296]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14080204800]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14180941824]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14164152320]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14197731328]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14130573312]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14147362816]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14214520832]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14264889344]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14248099840]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14298468352]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14332047360]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14231310336]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14348836864]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14281678848]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14365626368]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14315257856]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14399205376]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14382415872]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14449573888]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14432784384]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14499942400]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14415994880]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14466363392]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:03:55] openapi_requestor.py:316 [t:14483152896]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] base.py:89 [t:14466363392]: All tasks finished, exeutor will be shutdown\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13089624064]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13139992576]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13072834560]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13106413568]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13123203072]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13156782080]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13173571584]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13190361088]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13207150592]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13274308608]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13257519104]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13223940096]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13291098112]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13307887616]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13240729600]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13358256128]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13324677120]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13341466624]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13425414144]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13391835136]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13408624640]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13375045632]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13442203648]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13458993152]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13475782656]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13509361664]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13492572160]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13526151168]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13542940672]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13626888192]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13643677696]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13593309184]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13576519680]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13559730176]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13610098688]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13660467200]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13694046208]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13761204224]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13677256704]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13727625216]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13710835712]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13744414720]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13777993728]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13811572736]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13845151744]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13828362240]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13861941248]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13794783232]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14029836288]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13912309760]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13878730752]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13945888768]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13996257280]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13895520256]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13962678272]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14046625792]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14013046784]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14113783808]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14080204800]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13979467776]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:13929099264]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14096994304]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14130573312]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14164152320]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14063415296]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14147362816]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14180941824]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14197731328]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14214520832]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14281678848]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14231310336]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14264889344]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14248099840]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14298468352]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14332047360]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14315257856]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14432784384]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14348836864]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14382415872]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14399205376]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14466363392]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14415994880]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14365626368]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14499942400]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14449573888]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:05:49] openapi_requestor.py:316 [t:14483152896]: requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:07:43] base.py:89 [t:14499942400]: All tasks finished, exeutor will be shutdown\n"
     ]
    }
   ],
   "source": [
    "#-# cell_skip\n",
    "# 用户可以设置 service_model 为自己想要的模型名，来直接对数据进行批量推理，以 EB 4 为例\n",
    "result = ds.test_using_llm(service_model=\"ERNIE-Bot-4\")\n",
    "\n",
    "# 用户还可以设置 service_endpoint 来使用预置或自己的服务。\n",
    "result = ds.test_using_llm(service_endpoint=\"completions_pro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果用户有异步请求的需求，还可以使用 `atest_using_llm` 来进行异步批量推理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-02-23T07:49:35.322310Z"
    },
    "is_executing": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n",
      "[INFO] [02-23 15:49:35] openapi_requestor.py:357 [t:8596849280]: async requesting llm api endpoint: /chat/completions_pro\n"
     ]
    }
   ],
   "source": [
    "import asyncio\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()\n",
    "\n",
    "result = asyncio.run(ds.atest_using_llm(service_endpoint=\"completions_pro\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "拿到的 `result` 对象也是一个 `Dataset` 对象，可以继续使用千帆 Python SDK 进行后续处理，或者直接保存到本地。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-02-23T07:03:34.745795Z",
     "start_time": "2024-02-23T07:03:34.711794Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [02-23 15:03:34] dataset.py:934 [t:8596849280]: list local dataset data by 0\n",
      "[INFO] [02-23 15:03:34] dataset.py:560 [t:8596849280]: no destination data source was provided, construct\n",
      "[INFO] [02-23 15:03:34] dataset.py:358 [t:8596849280]: construct a file data source from path: output_file.csv, with args: {'is_download_to_local': False}\n",
      "[INFO] [02-23 15:03:34] file.py:165 [t:8596849280]: use format type FormatType.Csv\n",
      "[INFO] [02-23 15:03:34] dataset.py:253 [t:8596849280]: export as format: FormatType.Csv\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'prompt': '地球的自转周期是多久？', 'input_prompt': '地球的自转周期是多久？', 'llm_output': '地球自转是地球绕自转轴自西向东的转动，从北极点上空看呈逆时针旋转，从南极点上空看呈顺时针旋转。地球自转轴与黄道面成66.34度夹角，与赤道面垂直。关于地球自转的各种理论目前都还是假说。地球自转是地球的一种重要运动形式，自转的平均角速度为4.167×10-3度/秒，在地球赤道上的自转线速度为465米/秒。而地球的自转周期是**一天**，即**24小时**。\\n\\n如需更多与地球自转有关的信息，建议阅读天文类书籍或请教天文学专业人士。', 'expected_output': '大约24小时', 'request_complete_latency': 0.00042041699998662807}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(result.list(0))\n",
    "\n",
    "dataset_save_file_path = \"output_file.csv\"\n",
    "\n",
    "result.save(data_file=dataset_save_file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 对模型进行批量推理\n",
    "\n",
    "除了对 `Service` 进行批量推理，我们也可以对 `Model` 进行批量推理\n",
    "\n",
    "在对 `Model` 进行批量推理时，请先确认用到的数据集已经在千帆平台上发布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] [01-05 16:31:19] dataset.py:466 [t:8120441664]: no data source was provided, construct\n",
      "[INFO] [01-05 16:31:19] dataset.py:350 [t:8120441664]: construct a qianfan data source from existed id: 44478, with args: {'is_download_to_local': False}\n",
      "[INFO] [01-05 16:31:20] dataset.py:150 [t:8120441664]: a cloud dataset has been created\n",
      "[INFO] [01-05 16:31:20] dataset_utils.py:353 [t:8120441664]: start to create evaluation task in model\n",
      "[INFO] [01-05 16:31:24] dataset_utils.py:315 [t:8120441664]: start to polling status of evaluation task 2817\n",
      "[INFO] [01-05 16:31:24] dataset_utils.py:322 [t:8120441664]: current eval_state: Pending\n",
      "[INFO] [01-05 16:31:54] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:32:25] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:32:55] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:33:25] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:33:56] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:34:26] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:34:57] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:35:28] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:35:58] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:36:28] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:36:59] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:37:29] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:37:59] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:38:30] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:39:00] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:39:30] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:40:01] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:40:31] dataset_utils.py:322 [t:8120441664]: current eval_state: Doing\n",
      "[INFO] [01-05 16:41:01] dataset_utils.py:322 [t:8120441664]: current eval_state: DoingWithManualBegin\n",
      "[INFO] [01-05 16:41:01] dataset_utils.py:340 [t:8120441664]: get result dataset id 44629\n",
      "[INFO] [01-05 16:41:01] dataset.py:466 [t:8120441664]: no data source was provided, construct\n",
      "[INFO] [01-05 16:41:01] dataset.py:350 [t:8120441664]: construct a qianfan data source from existed id: 44629, with args: {}\n",
      "[INFO] [01-05 16:41:02] data_source.py:1096 [t:8120441664]: start to fetch dataset cache because is_download_to_local is set\n",
      "[INFO] [01-05 16:41:02] data_source.py:697 [t:8120441664]: no cache was found, download cache\n",
      "[INFO] [01-05 16:41:03] data_source.py:589 [t:8120441664]: get dataset info succeeded for dataset id 44629\n",
      "[INFO] [01-05 16:41:03] data_source.py:596 [t:8120441664]: start to export dataset\n",
      "[INFO] [01-05 16:41:03] data_source.py:600 [t:8120441664]: create dataset export task successfully\n",
      "[INFO] [01-05 16:41:05] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:06] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:08] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:08] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:10] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:11] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:13] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:13] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:15] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:15] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:17] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:18] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:20] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:20] data_source.py:610 [t:8120441664]: export succeed\n",
      "[INFO] [01-05 16:41:20] data_source.py:565 [t:8120441664]: get export records succeeded for dataset id 44629\n",
      "[INFO] [01-05 16:41:20] data_source.py:581 [t:8120441664]: latest dataset with time2024-01-05 16:41:19 for dataset 44629\n",
      "[INFO] [01-05 16:41:20] data_source.py:626 [t:8120441664]: start to download dataset zip from url https://bj.bcebos.com/easydata-upload/export_local/%E8%AF%84%E4%BC%B0%E4%BB%BB%E5%8A%A1_model_run_LzzKiuZ4Q0_%E7%BB%93%E6%9E%9C%E9%9B%86_f23d10V1_20240105_164103.zip?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-01-05T08%3A41%3A20Z%2F3600%2Fhost%2Fcb1ed680558c418f42272cbe1269b3a980db021e9140c4225eeb0d5c58eb34aa\n",
      "[INFO] [01-05 16:41:21] data_source.py:645 [t:8120441664]: download dataset zip to .qianfan_dataset_cache/36737/44629/1/bin.zip succeeded\n",
      "[INFO] [01-05 16:41:21] data_source.py:672 [t:8120441664]: unzip dataset to path .qianfan_dataset_cache/36737/44629/1/content successfully\n",
      "[INFO] [01-05 16:41:21] data_source.py:676 [t:8120441664]: write dataset info to path .qianfan_dataset_cache/36737/44629/1/info.json successfully\n",
      "[INFO] [01-05 16:41:21] data_source.py:725 [t:8120441664]: dataset cache is outdated, update cache\n",
      "[INFO] [01-05 16:41:22] data_source.py:589 [t:8120441664]: get dataset info succeeded for dataset id 44629\n",
      "[INFO] [01-05 16:41:22] data_source.py:565 [t:8120441664]: get export records succeeded for dataset id 44629\n",
      "[INFO] [01-05 16:41:22] data_source.py:581 [t:8120441664]: latest dataset with time2024-01-05 16:41:19 for dataset 44629\n",
      "[INFO] [01-05 16:41:22] data_source.py:596 [t:8120441664]: start to export dataset\n",
      "[INFO] [01-05 16:41:23] data_source.py:600 [t:8120441664]: create dataset export task successfully\n",
      "[INFO] [01-05 16:41:25] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:25] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:27] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:27] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:29] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:30] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:32] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:32] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:34] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:35] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:37] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:37] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:39] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:40] data_source.py:613 [t:8120441664]: export status: 1, keep polling\n",
      "[INFO] [01-05 16:41:42] data_source.py:605 [t:8120441664]: polling export task status\n",
      "[INFO] [01-05 16:41:42] data_source.py:610 [t:8120441664]: export succeed\n",
      "[INFO] [01-05 16:41:42] data_source.py:565 [t:8120441664]: get export records succeeded for dataset id 44629\n",
      "[INFO] [01-05 16:41:42] data_source.py:581 [t:8120441664]: latest dataset with time2024-01-05 16:41:40 for dataset 44629\n",
      "[INFO] [01-05 16:41:42] data_source.py:626 [t:8120441664]: start to download dataset zip from url https://bj.bcebos.com/easydata-upload/export_local/%E8%AF%84%E4%BC%B0%E4%BB%BB%E5%8A%A1_model_run_LzzKiuZ4Q0_%E7%BB%93%E6%9E%9C%E9%9B%86_f23d10V1_20240105_164122.zip?authorization=bce-auth-v1%2F50c8bb753dcb4e1d8646bb1ffefd3503%2F2024-01-05T08%3A41%3A42Z%2F3600%2Fhost%2Fe94091781afdfbb5dd8acbbd52697410f579db8959f78d5711eb0571823e9487\n",
      "[INFO] [01-05 16:41:43] data_source.py:645 [t:8120441664]: download dataset zip to .qianfan_dataset_cache/36737/44629/1/bin.zip succeeded\n",
      "[INFO] [01-05 16:41:43] data_source.py:672 [t:8120441664]: unzip dataset to path .qianfan_dataset_cache/36737/44629/1/content successfully\n",
      "[INFO] [01-05 16:41:43] data_source.py:676 [t:8120441664]: write dataset info to path .qianfan_dataset_cache/36737/44629/1/info.json successfully\n",
      "[INFO] [01-05 16:41:43] dataset.py:905 [t:8120441664]: list local dataset data by 0\n",
      "[INFO] [01-05 16:41:43] dataset.py:905 [t:8120441664]: list local dataset data by None\n",
      "[INFO] [01-05 16:41:43] dataset.py:905 [t:8120441664]: list local dataset data by 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'prompt': '人类的基本单位是什么？', 'input_prompt': '人类的基本单位是什么？', 'llm_output': '人类社会学界、群落、组织、团队、组织、机构、公司、政府、公司、家庭、家庭、军队、学校和家庭、村落、门派、门派、门派、门派、门派是人类的集合。\\n\\n\\n\\n\\n人类的基本结构层次单位是集合名词，人类的基本单位是集合名词，人类的基本单位是集合名词，人类的基本单位是集合名词，人类的基本单位是个人。\\n\\n\\n人类的基本单位是基本单位是个人。', 'expected_output': '人类'}\n"
     ]
    }
   ],
   "source": [
    "#-# cell_skip\n",
    "cloud_dataset_id = \"dataset_id\"\n",
    "\n",
    "qianfan_ds = Dataset.load(qianfan_dataset_id=cloud_dataset_id)\n",
    "\n",
    "result = qianfan_ds.test_using_llm(model_version_id=\"amv-qb8ijukaish3\")\n",
    "print(result[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 进阶能力\n",
    "\n",
    "在调用 `test_using_llm` 时，用户还可以传入一些额外参数来支持额外的功能，比如设置 Prompt 模板，设置人设字段，或者传入大模型调用时的超参数\n",
    "\n",
    "当对服务进行非对话类推理时，用户可以传入 `prompt_template` 参数来传递一个 Prompt 模板。`prompt_template` 是一个千帆 Python SDK 的 `Prompt` 对象，用户可以通过设置 `Prompt` 对象的 `template` 成员来自定义被用于推理的模板，模板渲染出来的内容将会被作为最终输入提交给大模型。以示例数据集为例，我们可以这么指定一个模板："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qianfan.common import Prompt\n",
    "\n",
    "prompt = Prompt(template=\"请你就以下问题进行回答: {prompt}\")\n",
    "\n",
    "# 传递给函数\n",
    "result = ds.test_using_llm(service_model=\"ERNIE-Bot-4\", prompt_template=prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除此之外，用户还可以传入 `system_prompt` 参数来指定对话中大模型需要遵守的人设"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = ds.test_using_llm(service_model=\"ERNIE-Bot-4\", system_prompt=\"人设 prompt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用户在进行批量推理时，还可以直接在 test_using_llm 中传入模型支持的超参数，例如我们可以这么设置模型的 `temperature` :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = ds.test_using_llm(service_model=\"ERNIE-Bot-4\", system_prompt=\"人设 prompt\", temperature=0.1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
